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In this paper, we develop a general approach for probabilistic estimation and optimization. 
An explicit formula is derived for controlling the reliability of probabilistic estimation based 
on a mixed criterion of absolute and relative errors. By employing the Chernoff bound and 
the concept of sampling, the minimization of a probabilistic function is transformed into an 
optimization problem amenable for gradient descendent algorithms. 
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Abstract 



1 Estimation of Probability 



CN ■ It is a ubiquitous problem to estimate the probability of an event. Such probability can be 

q^ ■ interpreted as the expectation, E[X], of a Bernoulli random variable X. More generally, if X is 

a random variable bounded in interval [0, 1] with mean E[X] = \x G (0, 1), we can draw n i.i.d. 

samples X±, ■ ■ ■ ,X n of X and estimate /i as p, = ' =1 — -. Since fi, is of random nature, it is 

crucial to control the statistical error. For this purpose, we have 

00 . 

Q . Theorem 1 Let e a € (0, 1) and e r € (0, 1) be real numbers such that — + e a < ~. Let S G (0, 1). 

Let X±,---X n be i.i.d. random variables defined in probability space (fl, =^,Pr) such that < 

K/. Xi<l and E[Xi] = /J, G (0, 1) for % = 1 • • • ,n. Let /x = ^'^ ' . Then, 

a' 



Pr^ l/z - //| < e a or 
provided that 



fi-H 



(i 



< e r \ > 1 - 5 (1) 



e r In ? 

n > ^^ -, 7 . (2) 

(e a + £ a £r) ln(l + e T ) + (e r - e a - e a e r ) In 1-^ 



It should be noted that conventional methods for determining sample sizes are based on 
normal approximation, see [4] and the references therein. In contrast, Theorem 1 offers a rigorous 
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method for determining sample sizes. In the special case that X is a Bernoulli random variable, 
a numerical approach has been developed by Chen [2] which permits exact computation of the 
minimum sample size. 

2 Optimization of Probability 

In many applications, it is desirable to find a vector of real numbers 9 to minimize a probability, 
p(0), which can be expressed as 

p{9) = Pr{Y(9,A) <0}, 

where Y(9, A) is piece-wise continuous with respect to 9 and A is a random vector. If we define 

MA,0)=E[e-^^)], 

then, applying Chernoff bound [3], we have 

p(0)<inf>(A,0). 

A>0 

This indicates that we can make p{9) small by making //(A, 6) small. Hence, we shall attempt to 
minimize //(A, 9) with respect to A > and 9. 

To make the new objective function /j,(X, 9) more tractable, we take a sampling approach. 
Specifically, we obtain n i.i.d. samples A\, ■ ■ ■ , A n of A and approximate fi(\, 9) as 

spn -\Y{e,Ai) 

g{\9) = ^^ . 

n 

A critical step is the determination of sample size n so that g(X, 9) is sufficiently close to //(A, 9). 
Since < e~ XY ^' > < 1, an appropriate value of n can be computed based on ([2]) of Theorem 1. 
Finally, we have transformed the problem of minimizing the probability function p(9) as the 
problem of minimizing a piece-wise continuous function g(X,9). Since g(\,9) is a more smooth 
function, we can bring all the power of nonlinear programming to solve the problem. An extremely 
useful tool is the gradient descendent algorithm, see, e.g. [1] and the references therein. 

3 Proof of Theorem 1 

To prove the theorem, we shall introduce function 

g(e, /i) = (fi + e) In \- (1 - /j, — e) In 



\x + s 1 — fJ, 

where < e < 1 — fi. We need some preliminary results. 
The following lemma is due to Hoeffding [5]. 



Lemma 1 

Pr{£t > fi + e} < exp(n g(e, fi)) for < £ < 1 — fi < 1, 

Pr{/2 < /U — e} < exp(?i g(— e, /i)) /or < s < fi < 1. 

Lemma 2 Lei < e < 2- Then, g(e,fi) is monotonically increasing with respective to fi € 
(0, 2 ~ e ) an d "monotonically decreasing with respective to fi € (5, 1 — e). Similarly, g(—e,fi) is 
monotonically increasing with respective to fi & (e, 5) an ^ monotonically decreasing with respective 
to fie (i + e, 1). 

Proof. Tedious computation shows that 



- In — r -\ h 



and 



d 2 g(e,[i) 



(fi + e)(l-fi) fi 1-fi 



dfi 2 fi 2 {fi + e) {l-fi) 2 {l-fi-e) 

for < e < 1 — fi < 1. Note that 

dqie.u) . , 1 — 2e 

a L=i = ln r" ^r + g<0 

a^ ^2 1 + 2e 



<0 



because 



Moreover, 



because 



l-2e 



In 1 -^ e 4- e 



(k 



I -As 2 



<0. 



dg(e,fi) t 



dfi M_ 2 



, 1 - 2e 4e 
_, = In — + — > 



1 + 2e 1 - 4e 2 



In 1 ~ 2e -1- 4e .. 



32e 2 



rfc 



(1-e 2 ) 2 



>0. 



Similarly, 



dg(s,fi) fi(l-fi + e) e e 

= in ■ 



and 



dfi 
d 2 g(-e,fi) 



(fi-e)(l-fi) ft 1-ft 



dfi 2 



fi 2 {fi-e) (l-^) 2 (l-^ + e ) 



<0 



for < e < fi < 1. Hence, 



because 



M_£^>, 1=ln i±^_ £>0 

9// '^-2 1 - 2e 



InM 2 ! _ e 
111 1-2e fc 

de 



l-4e 2 



>0; 



and 



as a result of 



dg(-e,fi) 
dfj, 



In 



l + 2e 



4e 



1 - 2e 1 - 4e 2 



<0 



m T=2i ~ T=IF 



32e 2 



ds 



(l- £ 2) 



2\2 



<0. 



Since 



9^ 1/^=2 



<0, 



9g(e.M) 



.i > and g(e,fi) is concave with respect to ii, it must be 



true that g(s, /x) is monotonically increasing with respective to /j, G (0, | — e) and monotonically 
decreasing with respective to ^u 6 (|,1 — e). Since gj L=l > 0) da lu=-+e ^ ^ an( ^ 
g(e, /x) is concave with respect to /x, it must be true that g(—s, li) is monotonically increasing with 
respective to /x G (e, |) and monotonically decreasing with respective to /x G (| +£, 1). 

□ 



Lemma 3 Let < e < 2- Then, 



g(e,IJ>) > g{-£,lj) V/xG ( e, - 



g(e, V) < g(~£, V) VjUG (-,!-£ 



Proof. It can be shown that 

d[g(e,/i) -g(-e,/*)] 

for < e < min(/i, 1 — /i). Note that 

£ 2 (l-2/x) 



In 



e 2 (l-2xx) 
(// 2_ £ 2 )(1 _^ 



and 



Therefore, 



and 



(^-, 2 )(l-^ >0 fOT £</i< 2 
e 2 (l-2/x) „ 1 

(/j2 _ £ 2 )(1 _ M)2 <o fi» e < 5 </*<i- e . 
gtgfogh^zfiM > o for £< ^<I 

<9e ^2 

%(^)- g Kri] <0 for e< i <AI<1 _ e . 

ae 2 



So, we can complete the proof of the lemma by observing the sign of the partial derivative 
a[ g ( E ,A0- fl (- e ,M)] and t he fact that g(e, /x) - g(-e, /x) = for e = 0. 

□ 



Lemma 4 £e£ < e < 1. Then, g (sli, /x) is monotonically decreasing with respect to ^ e (0, j^ 
Similarly, g(— e/x, /x) is monotonically decreasing with respect to fJ, G (0,1). 



Proof. Note that 

e 



dg{ew) = (1 + e)ln l-(l + *)/* _ (1 + e)ln(1 + g) + 

d 2 5 (e//, /x) e 2 



dfi 1 — /z 1 — /i 

and 

(9 2 o (eh,. u.) f 2 

<0 



V (l_ M) 2 [1 _ (1 + £)/i] 

for any /i G (o, ^ J . 

Since ag( ^' M) \^ =0 = e - (1 +e) ln(l + e) < 0, we have 



9/i \ 1+ £ 

and it follows that g (s/j,, /x) is monotonically decreasing with respect to /x G f 0, y^ 

Similarly, since 

dg(-£(i,(i) 



dfx 

and 



Im=o 



-e - (1 - e) ln(l - e) <0 



d 2 g(efi,n) 



2 

< 0, V/i G (0, 1) 



we have 

^(- £ ^)< , V M €(0,1) 



<9/x 
and, consequently, (7 (— £/i,/x) is monotonically decreasing with respect to /x G (0, 1). 



Lemma 5 Suppose < e r < 1 and < |^ + e a < ^ . Then, 

Pr{ju < /i - e a } < exp f n 5 f -e a , — j J 



for0<fi< 



□ 



(3) 



Proof. We shall show ([3]) by investigating three cases as follows. In the case of /i < e a , it is 
clear that 



Pr{£ < n - e a ] = < exp ( n g ( -e a , — j j . 



In the case of fj, = e , we have 

Pr{/i<Ai-ea} = Pr{£ = 0} = Pr{Xi = 0, t = 1, • ■ ■ ,n} 

n 

= ^Pr{X 4 = 0} = (Pr{X = 0})" 

= (l-Pr{X/0}) n < (l-E[X]) n 
= (l- A i)» = (l-e a )" 
= lim exp(n g(-s a , fj,)) 

( ( £ - 

< exp I n g I -e a; — 

where the last inequality follows from Lemma [2] and the fact that E a < ^ < 2 — £ a - 
In the case of e a < a < I s - , we have 

Er 

Pr{/1 < /i - e a } < exp(n g(-e a ,(i)) < exp (n g ( -e a , — )), 

where the first inequality follows from Lemma [1] and the second inequality follows from Lemma [2] 
and the fact that e a < ^r < \ — e a . So, is established. □ 

Lemma 6 Suppose < e r < 1 and < — + e a < ^ ■ Then, 

Piin>(l + e r )fi}<exp(ng(e a ,^)] (4) 

for ^ < n < 1. 

Proof. We shall show (jj]) by investigating three cases as follows. In the case of fi > jt^-, it is 
clear that 

Pr{/2 > (1 + e r )n} = < exp ( n g ( e Q , — J J . 

In the case of \i = j^- , we have 

Pr{/* > (1 + e p )m} = Pr{/2 = 1} = Pr{Xi = 1, z = 1, • ■ ■ ,n} 

n 

= ^Pr{X t = l} = (Pr{X = l}r 

2=1 

< [i n 



1 



lim exp(n g(e r fi, //)) 



/*- * 



l+er 

[ n# ( e a ,— 



where the last inequality follows from Lemma HI and the fact that — < t;ti — < ti — as a result 
of0< ^ r +e a <\. 

In the case of f 2 - < u < yj— , we have 

Pr{/2 < (1 + e r )/u} < exp(n g(e r fi, fi)) < exp (n g (e a , — j j , 

where the first inequality follows from Lemma [1] and the second inequality follows from Lemma 
SJ So, (gD is established. □ 



We are now in a position to prove the theorem. We shall assume ([2]) is satisfied and show that 
dl|) is true. It suffices to show that 

Pr{|/i -fi\> e a , |ju - n\ > e r /i} < 5. 

For < u < — , we have 

Pr{|/x - /i| > g a , \p.-fi\> e r n) = Pr{|/x - fi\ > e a ] 

= Pr{/x > n + e a } + Pr{£ < /i - e a }. (5) 

Noting that 0<// + e a <| iL + e a <^,we have 

Pr{/2 > /i + e a } < exp(n g(e a ,(J,)) < exp I n g I e a , — 

where the first inequality follows from Lemma [1] and the second inequality follows from Lemma 
[21 It can be checked that ([2]) is equivalent to 

e n \\ S 



^[ng^ a ,-))<-. 

Therefore, 

, S 
Pr{n >fi + e a }<- 

for < a < ^. 

On the other hand, since e a < |^ < 5, by Lemma [5] and Lemma El we have 

Pr{/x < /i - e a } < exp f n g I -s a , — J J < exp (n g (e a , — j j < - 
for < /x < |^. Hence, by fl5j), 

Pr{|£ - Ml > e a , \fi ~ i"| > ^r^} < 2 + 2 = ^ 
This proves flU for < /i < f 2 -. 



For — < Li < 1, we have 

Er ' 

Pr{|£ - fi\ > e a , \p,-fi\ > e r fj,} = Pr{\p, - fj,\ > e r fi} 

= Pr{£t > fi + e r fj,} + Pr{/i < fi — s r /j,}. 



Invoking Lemma [U we have 



Pr{/i > fi + e r ^} < exp ( n g ( e a , — 



On the other hand, 



Pr{/x < /x - e r fi} < exp(n g(-e r fi, /*)) < exp f n g ( -e a , — j j < exp f n g ( e Q , — j j 

where the first inequality follows from Lemma HJ the second inequality follows from Lemma HI 
and the last inequality follows from Lemma [3l Hence, 

Pr{|/x - fi\ > e , |/x-^| > e r Ai} < 2exp f nj I e , — j J < 5. 

This proves ([1]) for — < /x < 1. The proof of Theorem 1 is thus completed. 
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